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SUPERIMPOSING GRAPHIC REPRESENTATIONS OF GROUND 
LOCATIONS ONTO GROUND LOCATION IMAGES AFTER 
DETECTION OF FAILURES 
FIELD OF THE INVENTION 

The present invention relates to merging and displaying graphic 
representations of ground infrastructure locations with remotely detected material 
failures of the same ground locations. 

BACKGROUND OF THE INVENTION 

The surveillance of ground topography is well known in the art. In 
ground surveillance it is highly desirable to detect whether there has been a 
material failure in a man-made object such as a road, a pipeUne, an electrical grid, 
or other man-made structures of practical interest. When detected, a 
determination is made if remedial action must be taken. Often times a visual 
inspection is provided by a land-based crew that traverses an area by vehicle or 
foot, to determine if there is a material failure. In many cases it is required to 
inspect man-made structures frequently in order to determine the likehhood or 
development of potential material failures. In many cases these inspections are 
done by a ground site survey; individuals visit these locations and take 
measurements or other form of data on-site. Routinely the inspection crews carry 
a variety of maps drawings and other representations. This process becomes 
cumbersome, costly, inconvenient, and in many cases unreliable and xmsafe due to 
the danger of initiating investigatory or repair work on areas where buried cables 
or sewer lines may be present for example. It is frequently the case that an aircraft 
or a satellite includes an image capture device such as a charge coupled device 
(CCD). Airborne photographic systems can also be used for capturing images of 
adjacent areas. These images are then reviewed to determine if there is a material 
failure. Even though a failure in a man-made object can be identified, it is 
sometimes difficult to determine the location of that man-made failure relative to 
other landmarks or locations. 
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SUMMARY OF THE INVENTION 

It is an object of the present invention to provide an improved 
method for identifying the location of failures in man-made material objects 
shown in an image. 

5 This object is achieved by a method for superimposing graphic or 

symbohc representations of objects in ground locations onto visual images of such 
ground locations after detecting the presence of material failure(s) or failures in 
man-made structures in such ground locations comprising the steps of: 

(a) providing an image sensor spaced remotely from the ground 
O 10 and which sequentially captures a number of images of various ground locations 
f|| to provide digital images of such ground locations; 

? J (b) processing captured digital images to determine the 

presence of a potential material failure in a man-made structure in accordance 
" with predetermined coordinate positions which locate the man-made structures in 

^ 1 5 one or more of the captured digital images; 

iy (c) identifying reference points in the image of the ground 

f n locations correspondmg to the same reference points in the graphic representations 

of the ground location; and 

(d) superimposing the graphic or symbohc representation with 
20 the reference points onto at least one of the captured digital images. 

It is useful to combine the representation of material failures 
captured by a remote sensor with existing plans, maps, drawings or other 
representations in order to make the process of remediation more effective. The 
process of combining various representations requires image processing and 
25 digitization of the pre-existing maps plans or drawings in such a way that 

registration exists between the various representations is achieved. The term 
"registration" is used in analogy to the graphic arts to indicate the correct overiap 
of various layers of image information. 
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It is an advantage of the present invention to provide a more 
effective way of presenting information related to material failures in man-made 
structures by automatically processing images captured from a remote platform, 
identifying material failures in man-made structures, and by processing pre- 
5 existing ground location information in such a way that registration exists between 
various sets of information. This advantage will facilitate rapid and reliable 
identification of the locations of material failures in man-made structures and 
improve the time to bring about necessary repairs. 

BRIEF DESCRIPTION OF THE DRAWINGS 
10 FIG. 1 depicts a system for capturing images from an airborne or a 

Q satellite platform in accordance with the present invention; 

f =1 FIG. 2 is a flowchart in block diagram form of the process of 

:! capturing and processing images to detect potential material failures in man-made 

if! structures; 

15 FIG. 3 illustrates the combination of images and graphic or 

^ symbolic representations; 

Q FIG. 4 is a flowchart in block diagram form of an image processing 

^ algorithm which can be used in the system shown in FIG. 1 to identify material 

failures in man-made structures; 
20 FIG. 5 illustrates in block diagram form the process of combining 

images and graphic or symboUc representations; and 

FIG. 6 illustrates the capturing of an image, analysis to identify a 
potential material failure, combination with other representations, and 
communication over a channel to deliver information to and receive payment from 
25 a customer. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

FIG. 1 displays a sensor system 1 employed in the capturing of 
images in order to identify material failures in man-made structures. An image 
sensor spaced remotely from the ground sequentially captures a number of images 
30 of various ground locations; it may be located in a fixed structure or mounted on 
either an aerial or a satellite platform, for example. Images of the ground 
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containing various man-made structures such as roadways, pipelines, electrical 
power lines, agricultural, mining, real estate activity and the like are captured by 
this sensor system L The term "man-made structure" can also include other 
human activities such as insecticide spraying which after application can be 
5 detected by sensor system 1 . Sequential images may be captured in digital form 
and either stored in the aerial or satellite platform to be transferred later or 
transmitted via a radio link to a control ground station. The capture device 2 
includes an electronic sensor, typically a CCD or complementary metal oxide 
semiconductor (CMOS) imaging array that along with some imaging optics 

10 captures a picture of the scene in electronic form. In some instances a special 
optical filter 3 is attached to the input to the CCD or CMOS detector to filter the 
light wavelengths which are incident upon the detector. This optical filter 3 is 
chosen so as to maximize the signal-to-noise ratio for the detection of a specific 
type of material failure. Alternatively, the ground location image can be captured 

15 by conventional photographic cameras. Film images would then have to be 

converted to digital images by an image scanner that includes an image sensor. 
The system 1 also has an image capture control circuit 4 that sequences the 
operation of the capture device 2. As will be clear from FIG. 1, the operation of 
the various elements shown in system 1 are under the control of a control 

20 computer 3 1 . The image capture control circuit 4 controls capture device 2 and 
sends position and orientation information to a position and orientation storage 
circuit 5 with each captured image. Position information in the form of spatial 
coordinates can be provided by the customer in order to identify the location of 
man-made structures of interest. Such position information is also stored in 

25 position and orientation storage circuit 5. Position and orientation data are used 
along with predetermined coordinate positions to locate the man-made structures 
in the captured image. Control computer 3 1 causes image data to be stored in 
image storage 6 and can be processed to identify features of a scene in image 
processing circuit 7. The processing sequence is also directed by control 

30 computer 3 1 of the image data in this instance is to enhance the capability of the 
system 1 to identify material failures in man-made structures. The image 
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processing circuit 7 includes a storage memory (not shown) that includes a 
representation of different material failures to be detected and comparing the 
captured digital image with the material failures to determine the presence of a 
material failure, type of material failures and location of the material failures. 
5 With the exception of the capture device 2, the various elements of the system 1 
may be located either in the remote platform or at the ground station location. 
Moreover, many of the elements described can be embodied in software that can 
be understood to be within the control computer 31 . The capture device 2 is 
located in either the aerial or satellite platform or a fixed structure spaced above 
10 the ground. 

The overall process for detecting material failures in man-made 
structures is depicted in flowchart form in FIG. 2. The flowchart is in block 
diagram form and those skilled in the art will appreciate that many of the 
functions are controlled by the control computer 3 1 . The starting event includes 

15 initializing the capture device 2 and image storage 6 to erase any previously 

captured scene data. This first step is accomphshed in block 8. Next a new scene 
is captured in block 9 using the position information supplied by a customer 13 to 
trigger recording of the images. The image data along with position and time 
information necessary to identify the location and time of the current scene is 

20 stored in order to facilitate comparison with the same scene taken at other times. 
Image and other data are stored in a scene database 10 in order to perform such 
comparisons at a future time. Image analysis 1 1 is next performed in order to 
identify changes in the scene and facilitate identification of material failures in the 
man-made structures that appear in the scene. The latest scene image is compared 

25 with image data that has been previously stored in the scene database 10. If a 
material failure is not detected the process stops. Detection of a material failure 
may initiate further image analysis or the superimposition of graphic 
representations in block 12 as required by the customer 13. The identification and 
superimposition process finishes with the results of the analysis communicated to 

30 the customer 13. The communication make take many forms, for example a 
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telephone contact or e-mail notification of the detection of the material failure. 
The final step in the overall process is to correct the material failure. 

In many cases it is required to inspect man-made structures 
frequently in order to determine the likelihood or development of potential 
5 material failures or to initiate repairs of such structures. In many cases these 

inspections are done by a ground site survey; individuals visit these locations and 
take measurements or other form of data on-site. Routinely the inspection crews 
carry a variety of maps, drawings and other representations. This process 
becomes cumbersome, costly, inconvenient, and in many cases unreliable and 
kfc 1 0 unsafe due to the danger of initiating investigatory or repair work on areas where 

|3:?s. 

buried cables or sewer lines may be present for example. Routinely there exist 
[y graphic representations of ground locations like engineering drawings, plans or 

maps. These representations generally exist with specific scales, specialized 
Jl; symbols and legends that make them useful to interested parties. Examples are 

15 street maps, instrument surveys of land and buildings; surveys of the 

transportation grid, power grid, sewer lines, above ground and underground 
!jf telephone, electric, cable lines etc. These data in graphic representation or 

O sjmibolic form may be combined with image data identifying failures in man- 

made structures. 

20 FIG. 3 shows such a combination of images with graphic 

representations. In FIG. 3, graphic representations obtained from two separate 
plans, a power grid plan 14 and a water/sewer grid plan 15 are combined or 
superimposed with an image of a road captured remotely 16 to clearly depict the 
location of certain man-made structures and a failure in the roadway. Plan 14 

25 depicts the power grid plan illustrating the location of buried power cable while 
water/sewer grid plan 15 depicts the location of a manhole obtained from a 
water/sewer grid plan. These two features are identified and their location clearly 
marked in the image of the roadway 16 as graphic representations. A failure in 
the roadway 16 is also identified and marked in the image 16. The failure was 

30 detected and located by the methodology described above and illustrated in FIGS. 
1 and 2. 
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FIG. 4 depicts the algorithm used to process image data files from 
a database and identifies material failures if they have occurred. Two separate 
data files, scene (1) 17 and scene (2) 18, are made available for comparison. Both 
data files contain the same scene content, but they typically record images taken at 
5 different times. That is, the time between capturing the two images differs by a 
time At. Both image files or scenes undergo the process of orthorectification 19, 
that is, compensation for variations in position and angle at the time the scenes 
were recorded. This process is performed in order to allow an exact pixel by pixel 
comparison of the elements of a scene or image. It may or may not be necessary 
h'- 10 to correct the data in each scene for differences in the illumination 20 at the time 

each scene was recorded. Changes in the scene are identified in block 21 are used 
^ by the control computer 3 1 by detecting, using software, differences in the pixel 

%i content of the two scenes to be compared. Such changes may be reflected in the 

i ll intensity of the pixels, or in the shape of an object, corresponding to a finite 

f 15 collection of pixels. Such methods for identification of pixel or object changes are 
p well known to those skilled in the art. On the basis of such pixel changes the 

S«; material failure type is identified in block 22. 

O Following the steps of image capture and detection of material or 

potential material failures, the images are superimposed on existing graphic 

20 representations of the area in question where different forms of representation will 
serve to enhance the merged information. It is to be imderstood that the images to 
be superimposed can include previously captured digital images as well as current 
images of a scene. FIG. 5 depicts in somewhat greater detail the process of 
superimposing the graphic representation with the reference points onto at least 

25 one of the captured digital images to create a superimposition or overlay of 
graphic or symbolic representations with an image in which a fault has been 
identified. The image with the material failure identified 25 is processed to 
identify reference points in thescene 26. In parallel with this operation, customer 
coordinate data 24 is used to search the appropriate map database 27 for symbolic 

30 content that is relevant to the current area of concern. Reference points in the 

graphic representation 28 corresponding to the same reference points in the image 



of the ground location are also identified in the graphic representation or map 
selected from the map database 27. These common reference points 26 and 28, 
and the original customer coordinate data 24, are used in scaling and rotating 
operations 29 and 30, both the image with the identified material failure and the 
5 graphic representation or map. In this manner both the image and the graphic 
representation depict the same scene and are properly registered. A 
superimposition or overlay 32 is created wherein the important reference points 
and structures as commimicated by the customer 13, are shown in the image with 
the material failure identified and properly positioned. If additional overlays 32 

10 are required the process is repeated with a search of an appropriate map database 
and additional processing of the same type. In this manner multielement overlays 
can be created according to the customers specification regarding the type of 
structures that need to be identified. Such superimpositions or overlays can 
include further encoding with color or other symbols to identify predetermined 

15 areas of interest. The customer 13 may receive either layered information as a 
whole or in superimposable layers in either soft (electronic) display or printed 
hardcopy forms. Furthermore, the customer may be supplied combined sets of 
graphic representations including that an indication that there is potential material 
failure detected in a predetermined coordinate position. All of the various modes 

20 of delivery and variation in output type are depicted in block 13. 

FIG. 6 illustrates the entire process of capturing of an image, 
analysis to identify a material failure, creating the overlay, and communication 
over a computer network to deliver information to and receive payment fi-om the 
customer 13. A satellite 33 or an aerial platform 34 captures an image of a scene 

25 23 that contains a man-made structure (in this case electrical utility lines) to be 
analyzed. The image data is transmitted to a ground station 35 and transferred to 
the service provider's computer system 36. The image data is analyzed as 
previously described to determine whether a material failure has occurred and to 
create the overlays as requested by the customer 13. The analyzed image data is 

30 digitally superimposed with pre-existing graphic representations 14 and 15 of the 
ground location. The customer 13 for the service receives notification of the 



failure via a channel for example, a computer network such as the Internet, or via 
other means, such as telephony. The customer computer 37 receives the 
notification directly from over the computer network. The customer 13 subscribes 
to the service and pays for the service via the computer network. In this manner, 
the timely delivery of information regarding the status of a failure can be 
transmitted to the customer and the quality of service can be assured to be at a 
sufficiently high level. 

The invention has been described in detail with particular reference 
to certain preferred embodiments thereof, but it will be understood that variations 
and modifications can be effected within the spirit and scope of the invention. For 
example, the control computer 31 can itself be reprogrammed from a remote 
location and would include all the necessary communication links to permit such 
reprogramming. 
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PARTS LIST 



1 


sensor system 


2 


capture device 


3 


optical filter 


4 


capture control circuit 


5 


position and orientation control circuit 


6 


image storage 


7 


image processing circuit 


8 


initialization block 


M 9 


capture scene data, time and position information block 




scene database 




image analysis 


^3 12 


image analysis and create overlay 


s?i 13 


customer communication 


f 14 


power grid plan or map 




water/sewer grid plan or map 


i 16 


image of roadway captured remotely 


p 17 


image of scene at time t 


1 R 
i o 


linage oi scene ax iime i ^ zai 


19 


orthorectification 


20 


illumination correction 


21 


identify changed pixels block 


22 


identify fault type block 


23 


scene 


24 


customer coordinate data 
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Parts List cont'd 



25 


image with identified material failure 


26 


reference points 


27 


map database 


28 


reference points 


29 


scaling operation 


30 


rotating operation 


31 


control computer 


32 


overlay creation 


33 


satellite 


34 


aerial platform 


35 


ground station 


36 


service provider's computer system 


37 


customer computer 



